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Evaluation Reports 



This paper reports on the factors that characterize 
high-utilization evaluations. It i^; based on materials submitted to 
an American Educational Research Association (AERA) Division H 
competition for outstanding examples of evaluation utilization. The 
paper is organized into three sections. The first section outlines 
the background of the study: the purposes and procedures of the 
Division H competition, and the conceptual framework used in 
analyzing th6 data. The second section describes the analytic methods 
used, summarizes the results of the analysis, and specifies the six 
factors that seem to distinguish evaluations whose results are used 
by decision-makers. These factors are: level of evaluator effort, 
evaluator's leadership behavior, user involvement, evaluation 
reporting, evaluator involvement in implementing recommendations, and 
user commitment to use. The final section profiles the evaluators 
themselves. The data are viewed and interpreted from a 
multidisciplinary perspective which draws upon theories from 
psychology, sociology, organizational behavior, management, and 
marketing. (BW) 
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This paper reports on the factors that characterize 
high-utilization evaluations. It is based on materials submitted 
to an AERA Division H competition that was instigated and 
organized by the Evaluation Productivity Project of UCLA's Center 
for the Study of Evaluation (CSE). This project, which is funded 

•r 

by the National Institute of Education (NIE), has been at the 
forefront of research on evaluation utilization. 

The paper is organized into three sections. The first 
section outlines the background of the study: the purposes and 
procedures of the Division H competition, and the conceptual 
framework used in analyzing the data. The second section 
describes the analytic methods, summarizes the results of the 
analysis, and specifies the six factors that seem to distinguish 
evaluations whose results are used by decision-makers. The final 
section profiles the evaluators themselves. The data are viewed 
and interpreted from a multidisciplinary perspective which draws 

\ 

Upon theories from psychology, sociology, organizational 
behavior, management, and marketing. 

BACKGROUND 

The primary purpose of the Division H competition was to 
recognize and reward excellence in promoting evaluation use. Its 

ERIC 
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secondary purposes \were to make both evaluators and potential 
users more use-orieaied and to encourage practitioners to engage 
' in those kinds of behaviors that seem to promote evaluation 
utilization. 

Competition Procedures 

Evaluators entering the Division H competition were required 
to submit an application form (see Appendix A) in which they 
discussed those features of the evaluation that, in their 
judgement, had contributed significantly to its ultimate 
utilization. Evaluation utilization was verified by means of 
materials submitted by evaluation users, who were asked to 
. indicate the extent of use and to specify those aspects of the 
evaluator's performance that had contributed to use (see 
Verification Form in Appendix B). Over 30 users (administrators 
of the target programs) submitted such materials. 

Twelve evaluation studies were entered into the 
competition. While all of them dealt with educational programs, 
the majority (seven) were directed to public school programs at 
the district level. Of the remaining five, three addressed 
university programs; one, a statewide educational program; and 
one, a vocational *»ducation program for a special-needs 
population. Three of the twelve studies focused specifically on 
teacher effectiveness. Most of the evaluators had been retained 
as private consultants, although four were employees of the 
organizations housing the program being evaluated. 
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The review process involved over twenty judges, all of therr. 
Division H members, who were selected either" because they had 
contributed to the literature on evaluation use or because they 
were among the leading evaluation practitioners. Each evaluation 
study, along with the documents submitted as supporting evidence, 
was reviewed by three judges, who rated applicants on four 
dimensions: (1) conception of evaluation use, (2) extent of 
evaluation use, (3) degree of direct link between the evaluator 's 
efforts and subsequent use, and (4) uniqueness/creativity of the 
evaluator 's efforts to promote use (see Reviewer Rating Form in 
Appendix C). In addition, to ensure against rater bias, e€\ch 
rater ranked the applicants whom she or he had reviewed. Ratings 
were then totaled, and from these total scores, four finalists 
were identified; a fifth finalist who had received relatively 
high rankings was added to this group. The competition winner, 
and two honorable mentions, were chosen by a subset of raters at 
a full discussion session of the 1984 AERA convention. 

Conceptual Framework 

According to the definition employed in this paper, which 
was developed by Alkin (1975) and Patton et al. (1978), the term 
"use" is not limited to the direct application of evaluation 
findings to a specific decision but rather refers to the broader 
supporting role that evaluation findings often play in the 
complex, ongoing process of decision-making. As Alkin (1982) 
points out, the best definition for utilization is one that 
recognizes the gradual, incremental influence of evaluation. 



This view of utilization was further developed by CSE through a 
series of evaluation case studies (Alkin, Daiilak, & white, 
1979), an evaluator field study (Daiilak, 1980), and an 
administrator user survey (Sr.cher, Alkin, & Flesher, 1981). 

On the basis of these empirical data', Alkin and his 
colleagues (1985, in press) have developed a "utilization 
framework" that classifies the factors affecting evaluation use 
into three interrelated categories: human factors (evaluator and 
user characteristics)? context factors (pre-existing evaluation 
bounds, organizational features, and project characteristics); 
and evaluation factors (procedures, information dialogue, 
substance of information, and reporting), within each of these 
factors, specific elements that influence utilization have been 
identified. For example, critical evaluator characteristics 
include commitment to use,, rapport with users, and credibility. 
Crii:ical user characteristics include interest in the evaluation, 
commitment to use, and information-processing preferences. 
Pre-existing evaluation bounds, a context factor, encompasses 
written requirements, contractual obligations, and fiscal 
constraints. Evaluation reporting, an evaluation factor, 
includes the frequency and timing of reports and the mix of 
statistical and narrative data in reports. Many other elements 
are subsumed under the various factors (for a complete listing, 
see Alkin et al . (1985, in press)). 

: 0 



ANALYSIS OF APPLICANT DATA 

The comments of users and evaluators were analyzed by 
reading through the application and verification materials and /' 

abstracting all statements pertaining to use. These statements 

I 

were then classified according to the three categories of 
utilization factors: human, context, and evaluation. (Ur^r data 
are summarized in Appendix D, and evaluator data in Appendix E.) 
In some instances, the results confirmed the importance of the 
factors already specified 'in the utilization framework described 

above. In other cases, the results suggested that certain 

S 

factors not explicitly identified by previous research need to be 
further explored, with a view to elaborating the framework. 

Almost half of both the users' and the evaluators* comments 
pertained to human factors, and nearly all of the remainder 
referred to evaluation factors. Context factors were rarely 
mentioned, probably because the . competition emphasized the 
evaluator 's contribution to enhanced utilization. 

0 ' 

The most frequently cited evaluator characteristics were 
choice of role and willingness to involve users in the 
evaluation. The users* interpretation of choice of role is a 
unique one and will be discussed in more detail below. Also 
important were the evaluator 's credibility and rapport with 
users. 

The evaluation factors most frequently mentioned were 
procedures and reporting. Both users and evaluators recognized 
the importance of sound methodology, user-oriented designs, and 



follow-up procedures. Of the several reporting features cited as 
contributing to evaluation utilization, a good balance between 
statistical and narrative data was seen as most relevant. 

While the factors cited by evaluators tended to confirm the 
utilization framework already developed, users' comments 
suggested 'some additional factors that may influence 
utilization. Because of their significance from the users' 
perspective, .six factors merit further discussion for the insight 
they give into utilization: 

o Level of evaluator effort 

o Leadership behavior 

o User involvement 

o Evaluation reporting 

o Evaluator involvement in implementing recommendations 
o User commitment to use 

Level of Evaluator Effort 

Users frequently said that the exceptional level of effort 
which the evaluator put into the evaluation contributed to their 
utilization of the results. The following comments illustrate 
this point: 

The evaluator (E> has been willing to expend time and 
energy beyond the typical work day to do the necessary 
work . 

The sustained efforts of E over the last three years 
have insured that there is a broad commitment to this 
evaluation . 

E's drive to see the project through and ^willingness to 
reach out and help contributed to making the evaluation 
useful . 
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Clearly, users appreciate the effort put forth by the 
evaluator, not oi;ly because it results in a higher-quality 
evaluation and a better evaluation report but also because it 
signifies a commitment to the evaluation and a concern about its 
outcomes, it would seem, then, that when the evaluator manifests 
a high level nf energy, the likelihood of evaluation use 
increa^es« 

The equity theory of motivation (Adams, 1965; Weick, 1966) 
suggests^ tha.t something besides simple appreciation is at work 
here. This social comparison theory views human relationships as 
transactions involving inputs and outputs. In negotiating 
exchanges in the work environment, people seek a balance between 
inputs and outputs. According to this theory, the evaluator 's 
level of effort constitutes an input, and utilization of the 
recommendations advocated by the evaluator constitutes an 
output. Therefore, when level of evaluator effort is judged to 
be high, users are motivated to demonstrate high utilization. 

Leadership Behavior 

a' number of the evaluator characteristics noted by users as 
contributing to utilization can be characterized as leadership 
behaviors. And while some ©f these behaviors are already 
represented in the utilization framework (i.e., choice of role), 
i't seems reasonable to restructure the framework slightly by 
subsuming these elements under the "leadership behavior" rubric. 
Such an alternative conceptualization may provide additional 



insights into the dynamics of evaluation,' utilization. 

A sizable portion of the management literature deals with 
the definition of leadership and of what constitutes leadership 
behavior (see Fiedler, 1967; Hollander, 1978; House J. Baetz, 
1979; Katz & Kahn, 1978; Vroora & Yetten, 1973). Perhaps the most 
thorough of these discussions comes from social psychologists 
like Rensis Likert (1961) and Ralph Stogdill (1974), who have 



id€?ntified nine dimensions of leadership: initiation (originates 
new ideas and new practices), membership (mixes with the group), 
representation (acts in behalf of the group), integration 
(reduces conflicts between members), organization (structures the 
work of members), communication (provides information to members 
and shows awareness of - affairs pertaining to the group)* 
recognition (expresses approval of group members), production 



(sets levels of el^fort for greater achievement), and 

consideration (helps members and explains procedures). These 

dimensions were epitomized in$ the actions of evaluators, as 

reported in a number of statements made by users? 

E's information caused us to initiate actions that 
might never have been taken without his impetus. 
(Initiation) 

E has the ability to develop rapport and trust with key 
program personnel. This trust is reflected in the way 
thos# she interviews and surveys open up to her with 
total confidence. (Membership) 

The information E has been able to provide for us was 
juSt the documentation we needed to justify program 
decisions. (Representation) 

E worked with program personnel and upper 
administration to resolve conflicts, facilitating a 
process where all points of view were heard. 
( Integration ) 
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The evaluation report contained practical management 
plans based on the findings. (Organization) 



E's evaluation report was primarily useful in telling 
our story. He was capable of making explicit our 
commonly held (but taken for granted) assumptions about 
what we do and why we do it. (Communication) 

E has provided the impetus for the development of goals 
and objectives to strengthen and improve programs. 
( Production ) 

E provided the staff directly connected to the program 
. . . with much positive feedback. (Recognition) 

E explained what could and could not be measured, what 
instruments would be used and why. (Consideration) 



, User Involvement 



The evaluator 's willingness to involve users — already 

identified in the utilization framework as an important evaluator 

characteristic — deserves further examination because of the 

frequency with which it was mentioned, both by users and by 

evaluators, as a determinant of utilization. Users expressed the 

importance of their own involvement as follows: 

As superintendent, I was involved in the total process 
enough to follow through on the report. 

E developed 'ownership" from the beginning of the 
evaluation process so that participants actually 
thought of it as "their" evaluation. 

During planning, E reached out to program participants 
for their involvement, assuring broach-based ownership 
of the entire evaluative process. 



It makes intuitive sense that users will be /more likely to 
use information that they asked for or played a part in 
generating, but is there more to it than that? Participant 
management theory (Likert, 1967) maintains that user involvement 



is a critical component of effective management. Participant 
management is characterized by supportive relationships, group 
decision-making, group methods of supervision, and organizational 
objectives that reflect the needs and desires of all shareholders 
in the organization. According to its proponents, not only does 
this method of management make for better decisions, but it also 
guarantees that people will be more committed to carrying out 
these decisions. Evaluators who strive to build rapport with 
users, who involve users in the design of the evaluation and in 
data coH|ction, and who consider the information needs of all 
users are following the principles of participant management, 
whether they are aware of it or not. While this theory validates 
the inclusion of users in decision-making, it does not really 
explain why this technique is so powerful. 

One possible explanation is that involving users in the 
process of evaluation changes their attitudes about what the 
program should be and how it should operate. This new attitude 
is reflected in the recommendations that emerge from the 
evaluation. Thus, acting on the recommendations is consistent 
with their attitudes, whereas failure to use evaluation findings 
is likely to result in a state of cognitive dissonance 
(Festinger, 1957) which must eventually be resolved. User 
involvement is really a technique for attitude change that, once 
accomplished, motivates behavioral change. 
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Evaluation Reporting 

While the utilization framework identifies several elements 

of evaluation reporting — • notably, content of reports (Alkin, 

Oaillak, & White, 1979} and style (Brown, Braskamp, & Newman, 

1978) — as critical to utilization, users in the Division H 

competition noted another critical reporting element: 

thoroughness. This element is implicit in several of the 

utilization framework factors, being most closely related to 

substance of evaluation information, which includes information 

specificity. Nonetheless, users' views on the importance of the 

evaluator's thoroughness in reporting me'^it further examination. 

The following comments are illustrative of these views: 

The depth and thoroughness of the first-year evaluation 
enabled me to spot quickly and accurately the problems 
with the project so that they could be remedied the 
next year. 

The thoroughness and detail on what went well, what did 
not, and why, gave me a wealth of information and 
insight which I could utilize, in detail. 

This preference for detailed information, rather than just 
summaries and generalities, underscores an important point. 
Evaluators must demonstrate their thoroughness, but at the same 
time, they must know the individual lasers well enough to tailor 
their presentation of information to the cognitive styles and 
preferences of the users. The users quoted above obviously 
prefer that full and precise data be included within the report. 
Other users, with less of a need for fine detail, prefer a 
summary of evaluation highlights. Evaluators must satisfy both 
groups by providing comprehensive but readable reports along with 



concise executive summaries* They should at all times preserve 
the impression theit comprehensive data are available as back-up. 

Style and format are two dimensions of evaluation reporting 

specifically included within the utilization framework. Report 

style is the manner in which the evaluation "message" (i.e., the 

findings) is executed. Marketers have identified several 

successful message execution styles: slice~of-life , mood or 

image, technical expertise, scientific evidence, and testimonial 

evidence (Kotler, 1980). The users in this sample tended to 

prefer styles that were literate, conversational, warm, and 

down-to-earth rather than overly formal and jargonistic. One 

user commented: 

E had the ability to translate facts and figures and 
charts that would normally baffle the layperson's mind 
into interesting material. 



Another noted: 

All of the numbers, statistics, data were surrounded by 
literate prose with appropriate quotations from Alexis 
de Tocgueville. 

Format, another component of message execution, can also 
make a difference in message impact. Format elements include the 



size and dimensions of the report, and ifijs use of color, 
illustration, and other graphic elements, "^he importance of 
format was acknowledged by many users. The following comment is 
typical : 

E's reports have been exceptionally well accepted. Her 
formatting and special touches, such as attractive 
customized covers and pertinent cartoons, make the 
reading enjoyable and interesting. 



18 
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Evaluator Involvement in Implementing Recommendations 



Users* comments also focused on the evaluator *s active role 

i 

in the actual » hands-on implementation of the study 

recommendations. These evaluators went well beyond the 

(Conventional role of encouraging utilization -- they made certain 

of it. The following statements illustrate tJiis proactive role: 

E took leadership and responsibility during discussions 
and in preparing proposals and plans for the Board of 
Education. 

E has been ' remarkably successful in working with 
academic units throughout the campus in 
institutionalizing outcome information on an ongoing 
basis. 

E has held workshops for teachers and staff and is 
always available to help solve problems. 



Clearly # evaluator involvement in implementing 
recommendations may take many forms. In some instances, the 
evaluator conducts workshops on the findings as a step in 
potential implementation. Or the evaluator may interpret the 

is 

action implications of*^ particular recommendations. A similar 
phenomenon occurred in a study of Title I evaluators (Alkin, 
Stecher, & Geiger, 1982): One evaluator helped to attain 
utilization by "suggesting changes in the program, planning next 
year's workshop, developing a dissemination plan, and creating a 
meaningful attendance policy" (p. 2). In essence, this evaluator 
trained school district personnel to use data in making school 
decisions by giving them practice with the process. 



Sociological theory on the management of change provides 
insight into how post-report evaluator behavior influences use. 



Keen and Scott Morton (1978) hold that the change process 
consists of three stages: unfreezing, which serves to disturb 
the current stable equilibrium and introduces the need for 
change? moving, which involves striking out in a new direction; 
and refreezing, which requires integrating the change into 
existing behavioral frameworks to recreate a whole, natural 
entity. Management theorists (Katz & Kahn, 1978) claim that, 
while many change agents are successful in the first and second 
stages, they fail to realize the critical importance of the last 
stage. The evaluation process can be vievjred as the first stage, 
the evaluation recommendations as the second stage, and the 
implementation of recommendations as the third stage. It is nc 
wonder that evaluator involvement in this final stage of the 
change process characterizes high-use evaluations. 

User Commitment to Use 

User commitment to use, which was identified as an important 

user characteristic in all of our prior work, also turns out to 

be important in this study. Although users themselves did not 

often cite this factor — perhaps because they were asked to 

focus on the evaluator in their documentation — evaluators 

referred to it frequently, as the following comments show: 

The people involved were ready for the evaluation and 
the changes. 

The leadership of the Chancellor and the Provost . . . 
was one of the unique factors that contributed to my 
success in promoting the use of evaluation data. 
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To some extent, commitment to use is a personality 
characteristic pf/ users and thus is already established by the 
time the evaluator enters the scene. The marketing literature 
(Rogers, 1962) places consumers on a continuum, depending upon 
their willingness to adopt new products or new ideas: early 
adopters, early majority, late majority, and laggards. The users 
in this sample were clearly early adopters. Given their 
predisposition to try new ideas, they may have utilized any 
evaluation findings they were given. On the other hand, our 
earlier research (Alkin, Daillak, & White, 1979) shows that the 
actions of the evaluator can positively influence the users' 
predisposition to use. 



PROFILE OF A HIGH-UTILIZATION EVALUATOR 

* 

This study provides some insight into the characteristics 
and behavior of those evaluators whose work has a high 
probability of being utilized. High-utilization evaluators have 
personality traits — concern, warmth, patience, integrity, 
openness, tact, willingness to listen — that make them 
attractive to users from the outset. But they are also true 
leaders. Motivated by the desire to see their work utilized, 
they systematically enlist the participation of all potential 
users. They often violate the conventional image of the 
evaluation consultant by becoming involved in program operations 
and program improvement. In all of their leadership activities, 
however, these evaluators remain conscious of users' concerns and 
problems. They frequently assume the role of teacher, coaching 



program personnel in evaluation techniques. Their enthusiasm is 
not lost on users, who perceive such evaluators as investing a 
great deal of effort in the evaluation process and as being truly 
concerned about improving the program being evaluated. 

When preparing the evaluation report, high-utilization 
evaluators are careful to respond to the cognitive styles of the 
various users and to translate quantitative data into interesting 
information about the program. Once the evaluation report has 
been delivered, they continue their active involvement in the 
program by providing specific guidance in the implementation of 
the recommendations. 
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APPENDICES 



Appendix A Application Form 

AERA DIVISION H" 
AWARD FOR EXCELLENCE IN EVALUATION UTILIZATION 



Evaluator; 



Program evaluated :__ ' 



School district/Organization: 

Address : 



1. Briefly describe the program you evaluated. 

'3 



■ 4 



2. What use was made of the evaluation information which you produced? Please 
describe in detail ^nd provide documentation of use, if available. Po not 
send the evaluation report itself. 



\ 



3. What were the unique factors in this evaluation contributing to making use 
happen? 
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Appendix B Verification Form 

AERA DIVISION H 
AWARD FOR EXCELLENCE IN EVALUATION UTILIZATION 



Dear 



Your evaluator, , who worked on the evaluation 

of , ♦ * nominee for an AERA 

(name of program or project) 
award for excellence in the area of promoting evaluation use. We would appreciate 
your comments to assist our ^nel In judging the performance of this nominee on the 
above-noted program evaluation. 

1. In what way br ways was the evaluation information provided to you about the 
above program useful ? 



In your judgment, what aspects of the evaluator 's own performance during the 
evaluation helped make the evaluation useful to you? 



Mail this form directly to: Name:. 



Dr. Marvin C. Alkin Position: 

Graduate School of Education 

Center for the Study of Evaluation Date; 

UCLA \ 
Los Angeles, CA 90024 \ 



ERIC Please return by March 1, 1984 . 27 



Appendix C 

AERA DIVISION H 
AWARD FOR EXCELLENCE IN EVALUATION UTILIZATION 

REVIEWER RATING FORM 

Please rate the applicant's submission, including documentation and validation, on 
the following criteria. Use the corroborating evidence submitted in support of the 
# application to assist your judgments. 

1 . Conception of evaluation use : 

1 the applicant showed little or no understanding of evaluation use 

2 

3 the applicant showed an acceptable level of understanding of evaluation use 

_ ^ 

5 the applicant showed full understanding of evaluation use 

2 Degree of evaluation use ; 

1 no indication of evaluation use 

2 

3 some indication of evaluation us^ 

4 

5 strong indication of evaluation use 



3, Degree of direct link between evaluator efforts and subsequent use : 

1 no link was demonstrated between the evaluator 's efforts and subsequent use 

2 

3 a minimal link was demonstrated between the evaluator 's efforts and subsequent use 

4 

___ 5 a strong link was demonstrated between the evaluator 's efforts and subsequent use 



4. Uniquepess/creativitv of evaluator' s effort at promoting use : 

1 the evaluator showed no creativity or originality of effort 

2 

3 the evaluator showed a minimum of creativity or originality of effort 

4 

5 the evaluator showed a significant amount of creativity or originality of effort 



Appendix C (cont'd. ) 



Please rank the entries you have read 
Rank Entry No . 



Name of Reviewer 
Date completed 
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Appendix D 

Frequency of User Comments by Utilization Framework Categories' 



Category 

Human 
Factors 



Context 
Factors 



Evaluation 
Factors 



Factor 



Evaluator Characteristics 



Total Category Frequency 
Project Characteristics 

Total Category Frequency 
Evaluation Procedures 

Information Dialogue 



Substance of 

Evaluation Information 



Element 



Commitment to use 
Willingness to involve users2 
Choice of role — leadership 
Rapport with users 
Political sensitivity 
Credibility 

Background and identity - 
Level of perceived effort 



Innovativeness 



Methods 

Use of a general model 



Information relevance 
Information specificity 



Frequenc y 



3 
6 

13' 
7 
5 
9 
6 
6 

55 



18 
1 



Amount and quality of interaction 6 



9 
7 



o 

ERIC 



^These data are based on the comments of 34 users. 

'This variation of the existing factor was newly discovered in this research. 
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9 



4 



Appendix D 



Category Factor 

Evaluation Evaluation Reporting 
Factors 
(cont'd.) 



Total Category Frequency 

« 



Total Frequency 



:?2 

ERIC 



(cont'd. ) 
Element 



Frequency of information provided 

Timing of information 

Style of oral presentations 

Format of reports 

Mix of statistical/narrative data 



Appendix E 



Frequency of Evaluator Conunents by Utilization Framework Categories 



Category 

Human 
Factors 



Context 
Factors 



Evaluation 
Factors 



Factor 



Evaluator Characteristics 



User Characteristics 



Total Category Frequency 
Organizational Features 

Total Category Frequency 
Evaluation Procedures 

Information Dialogue 



Substance of 

Evaluation Information 



Element 



Willingness to involve users 
Choice of role 
Rapport with users 
Political sensitivity 
Credibility 



Interest in the evaluation 
Coiranitment to use 
Professional style 



External features 



Methods 

Use of a general model 



Amount of interaction 



Information relevance 
Information specificity 



Frequency 



7 
3 
3 
2 
2 



2 
4 
1 

24 



11 
1 



4 

1 



ERIC 



These data are based on the comments of 12 evaluators 
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Appendix E (cont'd.) 



Category 

Evaluation 
Factors 
(cont'd.) 



Factor 



Evaluiitlon Reporting 



Element 



Frequen6y of information provided 

Timing of information 

Style of oral presentation 

Format of reports 

Mix of statistical/narrative data 



Frequency 



2 
1 
1 
2 
3 



Total Category Frequency 



27 



Total Frequency 



53 



ERIC 



.77 



